Machine translation using neural network models

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for machine translation using neural networks. In some implementations, a text in one language is translated into a second language using a neural network model. The model can include an encoder neural network comprising a plurality of bidirectional recurrent neural network layers. The encoding vectors are processed using a multi-headed attention module configured to generate multiple attention context vectors for each encoding vector. A decoder neural network generates a sequence of decoder output vectors using the attention context vectors. The decoder output vectors can represent distributions over various language elements of the second language, allowing a translation of the text into the second language to be determined based on the sequence of decoder output vectors.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation (and claims the benefit ofpriority under 35 USC 120) of U.S. patent application Ser. No.16/521,780, filed Jul. 25, 2019, which claims the benefit of U.S.Provisional Application No. 62/703,518 filed Jul. 26, 2018, and titled“Machine Translation Using Neural Network Models,” which is incorporatedherein by reference.

BACKGROUND

This specification relates generally to machine translation using neuralnetwork models.

SUMMARY

Techniques related to machine translation are described below fortranslating text from one language to another language. The techniquesuse neural networks to translate one or more words in a first languageto a sequence of one or more words in a second language. For example,the neural networks can translate “What is your name?” in English to thesame phrase in Spanish—“¿Cómo to llamas?” The neural networks rely on anenhanced recurrent neural network machine translation model (RNMT+) thatis based on a recurrent neural network (RNN) model. The RNMT+ caninclude a deep learning artificial neural network with an encoder, amulti-headed attention mechanism, and a decoder. The multi-headedattention mechanism is situated in between the output of the encoder andthe input of the decoder. The encoder, multi-headed attention mechanism,and decoder are situated in a sequential fashion in order tosuccessfully produce a likelihood of a word in a second language basedon an input word in a first language.

The RNMT+ offers significant improvements over typical RNNs. Forexample, the RNMT+ improves translation accuracy over typical RNNsproducing similar translations. The RNMT+ can rely on variousoptimization techniques and processes that provide for a faster and moreefficient translation process during implementation. For example, theRNMT+ incorporates training processes that execute in parallel andoptimize the RNMT+ to outperform typical RNNs. As a result, a properlytrained RNMT+ can lead to significant improvement in translation speedand translation accuracy.

In some implementations, neural network machine learning models can beused to translate text from one language to another. Thesequence-to-sequence models discussed below can provide greater accuracythan prior approaches such as traditional RNN-based approaches and theTransformer model. Various techniques that provide enhanced accuracyinclude multi-headed attention, label smoothing, synchronous training,and layer normalization. In addition, hybrid architectures can alsoprovide further improvements to translation accuracy, by combiningstrengths of the techniques noted above with strengths of theTransformer model. For example, a decoder neural network can includemultiple decoder components having different neural network structures,in a serial or cascaded arrangement or in a parallel or multi-columnarrangement.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof obtaining, by the one or more computers, a series of feature vectorsrepresenting characteristics of the text in a first language;generating, by the one or more computers, encoding vectors from thefeature vectors by processing the feature vectors with an encoder neuralnetwork comprising a plurality of bidirectional recurrent neural networklayers, each encoding vector having a predetermined number of values;processing, by the one or more computers, the encoding vectors using amulti-headed attention module configured to generate multiple attentioncontext vectors for each encoding vector, wherein the multi-headedattention module includes multiple sets of parameters, and the multiplesets of parameters are respectively used to generate attention contextvectors from different subsets of each encoding vector; generating, bythe one or more computers, a sequence of output vectors using a decoderneural network that receives the attention context vectors, the decoderneural network comprising a plurality of unidirectional recurrent neuralnetwork layers, the output vectors distributions over various languageelements of the second language; and determining, by the one or morecomputers, a translation of the text into the second language based onthe sequence of output vectors.

Other embodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.A system of one or more computers can be configured to performparticular operations or actions by virtue of software, firmware,hardware, or any combination thereof installed on the system that inoperation may cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions.

The foregoing and other embodiments can each optionally include one ormore of the following features, alone or in combination. For example,one embodiment includes all the following features in combination.

In some implementations, the method includes storing data indicating thetranslation in a data retrieval system. The method includes accessingthe stored data indicating the translation. The method includesproviding the translation to one or more client devices over acommunication network.

In some implementations, the method includes for each of the pluralityof bidirectional recurrent neural network layers in the encoder neuralnetwork, the outputs of the forward layer and the backward layer areconcatenated before being fed into the next layer.

In some implementations, the method includes the plurality ofbidirectional recurrent neural network layers of the encoder neuralnetwork include long short-term memory (LSTM) layers.

In some implementations, the method includes the parameters of themulti-headed attention module are weighting values, and the multi-headedattention module applies the different sets of the parameters todifferent non-overlapping continuous chunks of the encoding vectors.

In some implementations, the method includes the multi-headed attentionmodule includes multiple chunk processors, each chunk processorincluding a separately trained neural network, each of the chunkprocessors generating a different one of the attention context vectorsfor each encoding vector.

In some implementations, the method includes the multi-headed attentionmodule generates the attention context vectors for a processing stepbased on (i) the encoding vector output by the encoder neural networkfor the processing step and (ii) a state of the first layer of thedecoder neural network.

In some implementations, the method includes the decoder neural networkis configured to receive the attention context vectors, concatenatedtogether, at each of the unidirectional recurrent neural network layersand at a softmax layer providing output of the decoder neural network.

In some implementations, the method includes the encoder neural networkand the decoder neural network include LSTM elements or gated recurrentunit (GRU) elements.

In some implementations, the method includes language elements of thesecond language include characters, word pieces, words, or phrases.

In some implementations, the method includes the encoder neural networkand the decoder neural network applies per-gate layer normalization foreach LSTM cell of the LSTM layers.

In some implementations, the method includes the encoder neural networkand the decoder neural network include a normalization layer betweeneach recurrent hidden neural network layer, the normalization layersconfigured to shift activations to a range that avoids saturation of asquashing function for propagation to a subsequent neural network layer.

In some implementations, the method includes the encoder neural network,multi-headed attention module, and/or the decoder neural network havebeen trained using synchronous training.

In some implementations, the method includes the encoder neural network,multi-headed attention module, and/or the decoder neural network havebeen trained using a learning rate that increases gradually over thecourse of training.

In some implementations, the method includes the encoder neural network,multi-headed attention module, and/or the decoder neural network havebeen trained using label smoothing that introduces variability intotarget labels.

In some implementations, the method includes label smoothing manipulatesan input vector for a neural network by altering or replacing one ormore elements of the input vector.

In some implementations, the method includes the encoder neural networkincludes a first encoder module and a second encoder module, wherein thefirst encoder module and the second encoder module have different neuralnetwork topologies. The first encoder module uses a transformer layerstructure and has layers that each include (i) a self-attention networksub-layer and (ii) a feed-forward network sub-layer. Additionally, thesecond encoder module includes a series of bidirectional recurrentneural network layers each providing normalization before processing bythe next recurrent layer.

In some implementations, the first encoder neural network (i) avoids anysequential dependences in the encoder network, performs computation foreach of the sub-layers having a computation sequence of normalization,transformation, dropout, and residual-addition, and (iii) theself-attention sub-layers are arranged so that each position in thecurrent layer has access to information from all other positions in theprevious layer.

In some implementations, the encoder neural network includes a cascadedencoder including one of the encoder modules providing output to anormalization layer that provides output to the other of the encodermodules. The output of the cascaded encoder is provided to the decoderneural network as the encoding vector.

In some implementations, the method includes the encoder neural networkincludes a multi-column encoder in which the first encoder module andthe second encoder module each generate an encoder output based on afeature vector, and the multi-column encoder merges the encoder outputsof the first encoder module and the second encoder module to produceencoding vectors.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of system for performing machinetranslation using a neural network model.

FIG. 2 illustrates an example of an enhanced recurrent neural networkmachine translation (RNMT+) model.

FIG. 3 illustrates an example of a multi-headed attention network.

FIG. 4 illustrates an example of a neural network model having anencoder that includes a transformer encoder module and an RNMT+ encodermodule in a cascaded arrangement.

FIG. 5 illustrates of a neural network model having an encoder thatincludes a transformer encoder module and an RNMT+ encoder module in aparallel or multi-column arrangement.

FIG. 6 illustrates a graph of a learning-rate schedule for a neuralnetwork machine translation model.

FIG. 7 shows an example of a computing device and a mobile computingdevice.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

Neural networks can be trained as machine translation models to converta sequence of one or more words in a first language to a sequence of oneor more words in a second language. For example, a machine translationmodel can convert the phrase in English “hello, how are you?” to thesame phrase in French “Salut, comment allez-vous?” A machine translationmodel with an encoder and decoder paradigm can be used to provideexcellent accuracy of translation. For example, the encoder and decoderparadigm can be implemented in an RNN model with a multi-headedattention mechanism acting in between the encoder and decoder.

As discussed further below, an improved RNN model can be used to predicta likelihood of a sequence of words using a deep learning artificialneural network. The RNN provides processed vectors between the encoderand decoder and is capable of capturing information from an infinitelength of input sentences. The RNN-based NMT approach, or more formallyknowns as RNMT, can incorporate parallel training to take advantage ofefficient computing devices, such as GPUs and Tensor Processing Units(TPUs), for example. By combining the RNMT with various modelingimprovements and optimization techniques, as described below, a RNMT+model can significantly outperform other NMT models in terms of accuracyof translations between various languages.

FIG. 1 illustrates an example of a system 100 for machine translationusing a neural network model. The system 100 includes a client device104, a translation server 106, an external provider 108, and a network110. The translation server 106 can include one or more computersconnected locally or over a network. The translation server 106 includesa feature extraction module 107, a machine translation model 109, and atranslation module 111. The feature extraction module 107 and themachine translation model 109 in the translation server 106 may beimplemented in software, hardware, firmware, or a combination thereof.

The example of FIG. 1 shows the translation server 106 processing aninput phrase 112 from a client device 104 of user 102. In the example,the translation server 106 shown in FIG. 1 receives the input phrase 112from client device 104 and outputs a translated phrase 114 back to theclient device 104. The phrase can be spoken (e.g., in the form of audiodata) or may be provided as textual data. The translation server 106processes the input phrase in a first language and outputs a translatedphrase in a second language, as further described below.

In other implementations, the translation server 106 can receive aninput phrase from the external provider 108 and output the translatedphrase to provide back to the translation server 106. The client device104 can transmit the input phrase 112 and receive the output-translatedphrase 114 over the network 110. Alternatively, the translation server106 can be implemented in the client device 104. Similarly, the externalprovider 108 can transmit an input phrase over the network 110 to thetranslation server 106. Additionally, the translation server 106 can beimplemented in the external provider 108.

In some implementations, the user 102 can input the phrase 112 to theclient device 104. For example, the user 102 can speak a phrase, such as“Hello, how are you?” to an application of the client device 104, andthe client device 104 converts the spoken phrase into data to transmit.In another example, the user 102 can type the phrase, e.g., “Hello, howare you?,” into the application of the client device 104 using atouch-screen or keyboard of the client device 104. The user 102 canenter the phrase in English or in a language other than English. Theapplication of the client device 104 can indicate to the user 102whether the other language can be properly translated by the translationserver 106.

In some implementations, the external provider 108 provides a text tothe translation server 106 for translation processing. The externalprovider 108 may provide text from a web page and request that the textbe translated into a specified language. As another example, theexternal provider 108 can be a television that provides second audioprogramming (SAP) to users that request SAP. The television can providespoken audio or subtitled text to the translation server 106. Thetranslation server can convert the spoken audio to text or process thesubtitled text (whichever the television sent) in a first language,convert the phrase in the first language to a phrase in a secondlanguage, and provide the converted phrase in the second language backto the television. In some cases, the external provider 108 can providethe phrase in a first language to the translation server 106 in realtime. In other cases, the external provider 108 can provide the phrasein a first language to the translation server 106 during an offlinemode.

The translation server 106 can include a feature extraction module 107,a machine translation model 109, and a translation module 111. Thefeature extraction module 107 generates feature data from the inputphrase data. The generated feature data can include a set of featurevectors, with each feature vector indicating textual or acousticcharacteristics during a different portion or window of the input phrasedata 112. Each feature vector can indicate properties of the phrasedata, such as segmentations of the phrase data 112. For example, eachfeature vector can include a letter of the phrase data, a word of thephrase data, or the entire phrase data itself. Additionally, eachfeature vector can include a portion of phrase data that describesprevious frames and/or subsequent frames. For example, as illustrated inFIG. 1, the feature extraction module 107 can extract feature vectorsfrom the input phrase data 112 “Hello, how are you?” to a first featurevector of “/Hel/,” a second feature vector of “/lo,h/,” a third featurevector of “/owa/,” a fourth feature vector of “/rey/”, and a fifthfeature vector of “/ou?/.” In other implementations, each feature vectorcan indicate tokenized words from the phrase data 112. For example, thetokenized words can include [“Hello”, “how”, “are”, “you”]. Thetranslation server 106 provides the extracted feature vectors as inputto the machine translation model 109.

The machine translation model 109 has been trained to outputprobabilities that indicate a likelihood of occurrence of translatedwords from the input phrase data 112, e.g., a distribution over apredetermined set of words in the second language for a text to betranslated into. The machine translation outputs can be a vector ofprobability scores that the respective likelihoods that different wordsshould be used in the translated sequence. In some implementations, anoutput vector or set of probabilities can be output from the machinetranslation model 109 for each feature vector provided as input.

In some implementations, the translation server 106 can include an inputsetting that indicates the desired input language for the phrase data112 and an output setting that indicates the desired output language forthe output phrase data 114. A user can change the input setting to adifferent language and can change the output setting to a differentlanguage. For example, user 102 may open an application or web page onhis/her client device 104 that allows for changing the language inputsetting and/or the language output setting at the translation server106. The user 102 can indicate to the translation server 106 to changethe input language from “English” to “French” and change the outputlanguage from “French” to “English.” Based on the user's input settingand output setting, the translation server 106 will select the propermachine translation model 109. For example, if the input setting is setto “French” and the output setting is set to “English,” the translationserver 106 will select the machine translation model 109 that properlytranslates French to English. Additionally, the translation server 106will select another machine translation model 109 if the input settingis set to “English” and the output setting is set to “French.” Thus, adifferent machine translation model 109 is required from a language pairand a direction of the language. In other implementations, one machinetranslation model 109 can be used for each of the language pairs and adirection of the language pairs.

In some implementations, the input phrase data 112 includes anindication of the language of the phrase data 112 and an indication ofthe desired language output. For example, the user 102 may indicate tothe client device 104 to translate the phrase “Hello, how are you?” fromEnglish to German. The client device 104 may package the phrase “Hello,how are you?,” data that indicates the input phrase is English, and datathat indicates the desired output phrase to be German into the. Upon thetranslation server 106 receiving the phrase data 112, the translationserver 106 can parse the phrase data 112 to determine the language ofthe input phrase. In some implementations, the translation server 106can automatically determine a language of the phrase itself. However,the phrase data 112 can still indicate the desired output language sothe translation server 106 can determine which machine translation model109 to use and further, determine the desired output language.

The machine translation model 109 outputs are provided to a translationmodule 111. The translation module 111 uses the sets of probabilityoutputs to identify a transcription in the desired language. Thetranslation module 111 can access a database of words at the translationserver 106 for identifying the transcription in the desired languagefrom the set of output probabilities. The translation module 111retrieves words from the database that indicate the highest likelihoodof probability to be the correct output in the desired output language.For example, the translation module 111 can determine that the correctoutput in the desired output language recites “Salut, commentallez-vous?” The translation module 111 may use a beam-search algorithmor other techniques to determine a transcription from the outputprobability scores from the machine translation model 109.

The translation server 106 provides the output phrase data 114 in thedesired output language to the client device 104 over network 110. Theoutput phrase data 114 can be provided to the client device 104 over thenetwork 110 in response to receiving the input phrase data 112 from theclient device 104.

FIG. 2 illustrates an example of an enhanced recurrent machine neuraltranslation model (RNMT+) 200. The machine translation model 109 caninclude the RNMT+ 200 as illustrated. The RNMT+ 200 may include a deepLSTM (Long Short Term Memory) recurrent neural network architecturebuilt by stacking multiple LSTM layers. The RNMT+ 200 includes anencoder 205, a multi-headed attention module 216, and a decoder 207. Theencoder 205 includes 6 bidirectional LSTM layers. The 6 bidirectionalLSTM layers in the encoder 205 includes a dropout layer 202, a forwardRNN layer 204, a backward RNN layer 206, a concatenation layer 208, adropout layer 210, and an add layer 212. The encoder 205's bidirectionalLSTM layers include a plurality of forward-propagating LSTM layers and aplurality of backward-propagating LSTM layers, with two LSTM layers ateach depth—one operating in the forward and another operating in thebackward direction in time over the input sequence. Both of these layersat the same depth are connected to both previous forward and backwardlayers. Finally, a projection layer 214 is added to the final output ofthe encoder 205.

As shown in FIG. 2, the translation server 106 provides a sourceembedding 201 into the encoder 205. The feature extraction modulegenerates a source embedding 201 (i.e., a feature vector) extracted fromthe input phrase 112. The source embedding 201 can be 1024 dimensionslong. Each layer in the encoder 205 and the decoder 207 can 1024dimensions. The dropout layer 202 included in the encoder 205 droppedone or more nodes in a neural network layer. For example, duringtraining of the RNMT+ 200, a random set of nodes in the dropout layer202 are not considered for processing during a forward or backward passof data. A benefit of including a dropout layer 202 in the encoder 505is that each node in that neural network layer will not becomeco-dependent upon one another. When nodes in a neural network layerbecome dependent upon one another, the individual value of each nodeleads to over-fitting of training data. In addition, by dropping thenumber of nodes in the dropout layer 202, the amount of training timerequired for RNMT+ 200 to converge nearly doubles, which ensures thateach node in the neural network layer is forced to sufficiently learnthe robust features of the training data, and thus, increases theoverall accuracy of the neural network output when used in practice. Theoutput of the dropout layer 202 includes a vector set of data thatincludes 1024 dimensions. The output of the dropout layer 202 isprovided as input to the forward RNN layer 204.

The forward RNN layer 204 is a type of recurrent neural network in whichdata is processed in the forward direction from left to right. Theoutput of the forward RNN layer 204 includes a vector set of data thatincludes 1024 dimensions. The output of the forward RNN layer 204 isprovided as input to the concatenation layer 208.

The concatenation layer 208 receives input from the output of thebackward RNN layer 206 and from the output of the forward RNN layer 204.The concatenation layer 208 takes in multiple inputs that have similarheight and width dimensions and concatenates the two inputs. Forexample, the output of the backward RNN layer 206 is a 1-row vector of1024 dimensions. Additionally, the output of the forward RNN layer 204is 1-row vector of 1024 dimensions. Thus, the output of theconcatenation layer 208 is a 1-row vector of 2048 dimensions from theconcatenated input from the forward RNN layer 204 and input from thebackward RNN layer 206.

The backward RNN layer 206 is a type of recurrent neural network inwhich data is processed in the backward direction, e.g., from right toleft or from the end of the sequence toward the beginning. The backwardRNN layer 206 at the input layer receives a feature vector from thesequence, but in the opposite order as the forward layer. For example,if there are n feature vectors in the input sequence, at the first timestep the forward layer receives input vector 1 while the backward layerreceives vector n. At the second time step, the forward layer receivesinput vector 2 while the backward layer receives vector n−1, and so on.The output of the backward RNN layer 206 includes a vector set of datathat includes 1024 dimensions. The output of the backward RNN layer 206is provided as input to the dropout layer 210.

The dropout layer 210 is similar in functionality to the dropout layer202. Dropout layer 210 receives a vector of 2048 dimensions from theconcatenation layer 208.

The add layer 212 sums together the output from the dropout layer 210and the output from the backward RNN layer 206. For example, the addlayer 212 can combine the vector from the dropout layer 210 and thevector from the backward RNN layer 206.

The projection layer 214 converts the output of the add layer 212 to adimensionality that matches the dimensional input requirements of thedecoder 207. For example, if the output of the add layer 212 is a 2×3matrix, and the input to the decoder 207 requires a 1 dimensionalvector, then the projection layer 214 converts the 2×3 matrix to a 6×1vector. After conversion, the projection layer 214 provides the 6×1vector to the multi-headed attention module 216.

The RNMT+ 200 also includes a multi-headed attention module 216. Themulti-headed attention module 216, which will be further described withrespect to FIG. 3, divides the vector from the output of the projectionlayer 214 by the number of neural networks in the multi-headed attentionmodule 216. For example, the multi-headed attention module 216 caninclude four neural networks for processing. If the input vector has a 1row vector with 1024 dimensions, then the multi-headed attention module216 divides the 1024 dimension by four (e.g., for the number of neuralnetworks). The multi-headed attention module 216 provides one fourth ofthe 1024 dimension vector, or one of the divided vectors, to each of itsneural networks for processing. By performing multi-headed attentioninstead of single headed attention, the vectors can be processed by eachneural network in parallel and as a result, the various dimensions fromeach of the divided vectors can be processed independently. Thus, theaccuracy of the multi-headed attention 216's processing is increased.

The RNMT+ 200 includes a decoder 207 with 8 unidirectional (forward)LSTM layers. Each unidirectional LSTM layer of the decoder 207 includesforward RNN layer 222 and 7 forward RNN layers 226. After the dropoutlayer 202, there is a structure comprising a concatenation layer 224, aforward RNN layer 226, a dropout layer 228, and an addition layer 230.This structure is repeated seven times as illustrated by the “7×” inFIG. 2. The output of the repeated layer structure is provided to asoftmax layer 232, which generates the output of the decoder, e.g.,mapping output activations of the network to a distribution ofprobability scores of the set of possible output words in the targetlanguage. In general, dropout layer 220 is similar in functionality todropout layer 202. Forward RNN layer 222 is similar in functionality toforward RNN layer 204. Concatenation layer 224 is similar infunctionality to concatenation layer 208. Forward RNN layer 226 issimilar in functionality to forward RNN layer 204. However, forward RNNlayer 226 receives a vector array that is two times the dimensionalitysize of the input provided to the forward RNN layer 204. For example,forward RNN layer 226 receives a vector array with a size of 1×2048, andforward RNN layer 204 receives a vector array with a size of 1×1024. Thedropout layer 228 is similar in functionality to the dropout layer 210.The addition layer 230 is similar in functionality to the add layer 212.

In some implementations, the translation server 106 uses targetembeddings 218, in sequences of training data representing accuratetranslations of full language sequences, to train the RNMT+ 200. Thetranslation server 106 can retrieve sub-word units for eachsource-target language pair for a full language sequence, such as asentence or phrase, from a database. The database can include sub-wordunits for various source target language pairs that includes, French toEnglish, English to French, English to German, and German to English, toname a few examples. For example, the sub-word units can for varioussource-target language pairs can include “Hello” to “Bonjour”, “Bonjour”to “Hello”, and “Hello” to “Guten Tag.” In another example, the fulllanguage sequence can include “Hello. How are you?” to “Bonjour. Commentallez-vous?” and each word in the full language sequence can be asub-word unit. The size of the database can be 32K sub-word units orgreater.

During training, the translation server 106 retrieves the source targetlanguage pairs from the database for training a particular model. Forexample, for training the RNMT+ 200 for an English to Frenchtranslation, the translation server 106 retrieves English to Frenchtranslation pairs to train the RNMT+ 200. The English to Frenchtranslation pairs can include “Hello” to “Bonjour,” “Goodbye” to “Aurevoir,” and “smile” to “sourire,” to name a few examples. Then, totrain, the translation server 106 first provides each source targetlanguage pair to the feature extraction module 107 to generate a featurevector embedding for each source and each target pair, respectively. Thetranslation server 106 provides the feature vector embedding for theword “Hello” as the source embedding 201 into the encoder 205.Simultaneously, the translation server 106 provides the feature vectorembedding for the word “Bonjour” as the target embedding 218 into thedecoder 207. The translation server 106 provides each of the sourcetarget language pairs of English to French from the database fortraining the English to French RNMT+ 200 model. Once the English toFrench RNMT+ 200 model converges and outputs probability scores 234 thatare high, the training is complete. The same process occurs for otherlanguage pairs. For example, the translation server 106 provides each ofthe source target language pairs of French to English from the databasefor training the French to English RNMT+ 200 model. The samefunctionality occurs for the English to German and German to Englishmodels. In other implementations, the translation server 105 retrieves afull language sequence from the database for training a particularmodel. The full language sequence can be in English, French, or German.The full language sequence can include a set of source sub-word unitsand target sub-word units. The translation server 106 can provide thefeature vector of each source sub-word unit from the full languagesequence and each target sub-word unit from the full language unit intothe encoder 205 and the decoder 207, respectively.

The softmax layer 232 is the final layer in the RNMT+ 200. Typically,the softmax layer 232 outputs a probability distribution of a number ofpossible outcomes from the inputs. The higher the probability outputfrom the softmax layer 232, the higher the likelihood of an indicationthat the translated output is correct. For example, the translationserver 106 may provide the input [“Hello”; “how”; “are”; “you”] asfeature vectors to the source embedding 201 input of the RNMT+ 200. TheRNMT+ 200 can translate the input phrase and output the translatedphrase in French with corresponding probability scores, such as,[“Salut”, 0.9; “comment”, 0.8; “allez”, 0.7; “-vous”, 0.9]. In someimplementations, the RNMT+ 200 model can process the phrase input 112 inEnglish to other languages, such as German, Spanish, and Russian, toname a few examples, based on how the model is trained. Alternatively,the RNMT+ 200 can process the text input 112 in another language, otherthan English, such as, Italian, German, and French, to name a fewexamples.

FIG. 3 illustrates an example of a multi-headed attention network. Themulti-headed attention module 216 can include a divider module 304, Nnumber of neural networks, and a concatenation module 318. FIG. 3illustrates three neural networks (e.g., neural network 312, neuralnetwork 314, and neural network 3) each trained with various weights andvarious parameters to process a portion of the N-dimensional vector 302.Any number of neural networks can be shown in the multi-headed attentionmodule 216, three are shown for illustrative purposes only. In someimplementations, the multi-headed attention module 216 helps the RNMT+200 align and translate the incoming feature vectors. The multi-headedattention module 216 aligns to identify which part of the input sequenceis relevant to each word in the output sequence. Additionally, themulti-headed attention module 216 translates by using relevantinformation of the input and weights of the neural network to select theappropriate output.

The projection layer 214 outputs an N-dimensional vector 302 (e.g.,encoder output vector) from the encoder 205. The projection layer 214may provide the N-dimensional vector 302 to have a dimension that equalsthe input dimension of the decoder 207. In some implementations, theoperation of the attention network may adjust the dimensionality byproducing a vector that has more or fewer values than the output of theprojection layer 214.

In some implementations, the projection layer 214 may output the vector302 to be a 1-row vector of 1024 dimensions. The translation server 106provides the 1-row vector of 1024 dimensions to a divider module 304 ofthe multi-headed attention module 216. The divider module 304 dividesthe N-dimensional vector 302 into a predetermined number of segments,with each of neural networks 312, 314, 316 receiving one segment. Thesegments are generally non-overlapping, contiguous groups of the encodedvector from the encoder module. For example, if the size of theN-dimensional vector 302 is 1×4096 and the number of neural networks isfour, then the divider module 304 divides the N-dimensional vector 302into four N-dimensional vectors of size 1×1024. As illustrated in FIG.3, there are three attention “heads,” e.g., three neural networks in theattention module, so the divider module 304 produces three outputvectors, vector 306, vector 308, and vector 310, each including adifferent portion of the encoder output vector 302. Vectors 306, 308,and 310 are each have a number of values equal to the number of valuesin the N-dimensional vector 302 divided by three (e.g., the number ofneural networks), so each vector 306, 308, and 310 is a different thirdof the vector 302.

The neural networks 312, 314, and 316 can be an RNN, a ConvolutionalNeural Network (CNN), or any other type of neural network.

In some implementations, the output of each neural network (e.g., neuralnetwork 312, neural network 314, and neural network 316) can be a set ofprobabilities. In other implementations, the output of each neuralnetwork can be a set of vector outputs. The size of the vector outputscan be the same size as each of respective vector 306, vector 308, andvector 310.

In some implementations, the concatenation module 318 combines theoutput from each of the neural networks into a data output vector. Asshown in FIG. 3, the concatenation module 318 concatenates the output ofthe neural network 312 with the output of the neural network 314. Theoutput of neural network 316 is concatenated with the output of neuralnetworks 312 and 314. For example, the neural network 312 can output avector of size 1×1048, the neural network 314 can output a vector ofsize 1×1048, and the neural network 316 can output a vector of size1×1048. The concatenation module 318 can concatenate in the followingway: [first vector|second vector|third vector]. The output of theconcatenation module 318 is provided to the softmax layer 232, theconcatenation layer 224, and the forward RNN layer 222 at the decoder207.

FIG. 4 illustrates an example of a system 400 of a transformer model andRNMT+ model components in an encoder. The system 400 illustrates acascaded encoder 404 and an RNMT+ Decoder 412. The RNMT+ Decoder 412 maybe similar to the decoder 207. The cascaded encoder 404 includes anRNMT+ Encoder 406, a layer normalization 408, and a transformer encoder410. The cascaded encoder 404 and the RNMT+ Decoder 412 can include amulti-headed attention module between the transformer encoder 410 andthe RNMT+ Decoder 412, similar to the multi-headed attention module 216.The focus of the system 400 is to enrich the set of statefulrepresentations by cascading a feature extractor with the cascadedencoder 404 and fine tuning the layers of the transformer encoder 410stacked on top of a pre-trained frozen RNMT+ encoder. The sourceembeddings 402 can be provided to the cascaded encoder 404 to start thetranslation process.

FIG. 5 illustrates an example of another system 500 of a transformermodel and RNMT+ model components in an encoder. The system 500 includesa multi-column encoder 504 and an RNMT+ Decoder 512. The RNMT+ Decoder512 may be similar to the decoder 207 and decoder 412. The multi-columnencoder 504 includes a Transformer Encoder 506, an RNMT+ Encoder 508,and a merge layer 510. The multi-column encoder 404 and the RNMT+Decoder 412 can include a multi-headed attention module between themerge layer 510 and the RNMT+ Decoder 512. The focus of the system 500is to merge the output of several independent encoders into a singlecombined representation (e.g., using the merge layer 510). The benefitof having a system 500 allows a further distinction as to whether theRNMT+ decoder 512 can distinguish information from the transformerencoder 506 and the RNMT+ encoder 508.

FIG. 6 illustrates a graph 600 of a RNMT+ learning-rate schedule. Asillustrated in the graphical FIG. 600, the RNMT+ encoder and decoder istrained with 1024 LSTM nodes. The input embedding dimension is 1024. Themulti-head attention module 216 has four neural networks. As can be seenin the graphical FIG. 600, the learning rate rapidly increases thenlevels off within the first 0.2 training steps. After 0.4 trainingsteps, the learning rate steadies off until flattening.

In many instances, new architectures are accompanied by a novel set oftechniques for performing training and inference that have beencarefully optimized to work in concert. This ‘bag of tricks’ can becrucial to the performance of a proposed architecture, yet it istypically under-documented and left for the enterprising researcher todiscover in publicly released code (if any) or through anecdotalevidence. This is not simply a problem for reproducibility; it obscuresthe central scientific question of how much of the observed gains comefrom the new architecture and how much can be attributed to theassociated training and inference techniques. In some cases, these newtechniques may be broadly applicable to other architectures and thusconstitute a major, though implicit, contribution of an architecturepaper. Clearly, they need to be considered in order to ensure a faircomparison across different model architectures.

The techniques and methods described below contribute significantly tothe success of recent architectures, namely ConvS2S and Transformer, andexplore applying these methods to other architectures, including RNMTmodels. In doing so, the enhanced version of RNMT, referred to as RNMT+significantly outperforms all individual architectures in the setupdescribed below. The new architectures introduced illustrate differentcomponents borrowed from RNMT+, ConvS2S and Transformer. In order toensure a fair setting for comparison, all architectures were implementedin the same framework, use the same pre-processed data and apply nofurther post-processing as this may confound bare model performance.

The contributions illustrated below are three-fold: (1) In ablationstudies, the effect of several modeling improvements (includingmulti-head attention and layer normalization) are quantified as well asoptimization techniques (such as synchronous replica training andlabel-smoothing), which are used in recent architectures. Thesetechniques are applicable across different model architectures. (2)Combining these improvements with the RNMT model into a new RNMT+ model,which significantly outperforms all fundamental architectures on thewidely-used WMT′ 14 English to French and English to German benchmarkdatasets. A detailed model analysis is provided along with a comparisonof RNMT+, ConvS2S and Transformer in terms of model quality, model size,and training and inference speed. (3) Inspired by the relative strengthsand weaknesses of individual model architectures, new modelarchitectures are composed that combine components from the RNMT+ andthe Transformer model, and achieve better results than both individualarchitectures

Two prior works provided empirical solutions to the difficulty oftraining NMT architectures (specifically RNMT). In one instance,elements of NMT architectures were explored as to which elements of NMTarchitectures have a significant impact on translation quality. Inanother instance, three specific techniques were recommended forstrengthening NMT systems and empirically demonstrated how incorporatingthose techniques improve the reliability of the experimental results.

In this section, the commonly used NMT architectures are discussed. RNMTmodels are composed of an encoder RNN and a decoder RNN, coupled with anattention network. The encoder summarizes the input sequence into a setof vectors while the decoder conditions on the encoded input sequencethrough an attention mechanism, and generates the output sequence onetoken at a time.

The most successful RNMT models consist of stacked RNN encoders with oneor more bidirectional RNNs, and stacked decoders with unidirectionalRNNs. Both encoder and decoder RNNs consist of either LSTM or GRU units,and make extensive use of residual or highway connections.

In Google-NMT (GNMT), the best performing RNMT model on the datasetsconsidered, the encoder network consists of one bi-directional LSTMlayer, followed by 7 unidirectional LSTM layers. The decoder is equippedwith a single attention network and 8 unidirectional LSTM layers. Boththe encoder and the decoder use residual skip connections betweenconsecutive layers.

The GNMT is adopted as the starting point for the proposed RNMT+architecture, following the public NMT codebase.

In the most successful convolutional sequence-to-sequence model, boththe encoder and decoder are constructed by stacking multipleconvolutional layers, where each layer contains 1-dimensionalconvolutions followed by a gated linear units (GLU). Each decoder layercomputes a separate dot-product attention by using the current decoderlayer output and the final encoder layer outputs. Positional embeddingsare used to provide explicit positional information to the model. Thegradients of the encoder layers are scaled to stabilize training. Theresidual connections across each convolutional layer are used and weightnormalization is applied to speed up convergence. The public ConvS2Scodebase is followed in the below experiments.

The Transformer model is motivated by two major design choices that aimto address deficiencies in the former two model families: (1) UnlikeRNMT, but similar to the ConvS2S, the Transformer model avoids anysequential dependencies in both the encoder and decoder networks tomaximally parallelize training. (2) To address the limited contextproblem (limited receptive field) present in ConvS2S, the Transformermodel makes pervasive use of self-attention networks so that eachposition in the current layer has access to information from all otherpositions in the previous layer.

The Transformer model still follows the encoder-decoder paradigm.Encoder transformer layers are built with two sub-modules: (1) aself-attention network and (2) a feed-forward network. Decodertransformer layers have an additional cross-attention layer sandwichedbetween the self-attention and feed-forward layers to attend to theencoder outputs.

There are two very important details to the model's performance: (1)Each sub-layer in the transformer (i.e. self-attention, cross-attention,and the feed-forward sub-layer) follows a strict computation sequence:normalize→transform→dropout→residual-add. (2) In addition to per-layernormalization, the final encoder output is again normalized to prevent ablow up after consecutive residual additions.

The latest version of the Transformer model in the public Tensor2Tensorcodebase is followed.

From a theoretical point of view, RNNs belong to the most expressivemembers of the neural network family. Possessing an infinite Markovianstructure (and thus infinite receptive fields) equips RNNs to modelsequential data, especially natural language effectively. In practice,RNNs are notoriously hard to train, confirming the well-known dilemma oftrainability versus expressivity. Convolutional layers are adept atcapturing local context and local correlations by design. A fixed andnarrow receptive field for each convolutional layer limits theircapacity when the architecture is shallow. In practice, this weakness ismitigated by stacking more convolutional layers (e.g. 15 layers as inthe ConvS2S model), which makes the model harder to train and demandsmeticulous initialization schemes and carefully designed regularizationtechniques.

The transformer network is capable of approximating arbitrary squashingfunctions, and can be considered a strong feature extractor withextended receptive fields capable of linking salient features from theentire sequence. On the other hand, lacking a memory component (aspresent in the RNN models) prevents the network from modeling a statespace, reducing the RNN's theoretical strength as a sequence model, thusthe RNN requires additional positional information (e.g. sinusoidalpositional encodings).

Above theoretical characterizations will drive explorations in thefollowing sections.

The models are trained on the standard WMT′14 English to French andEnglish to German datasets that comprise 36.3M and 4.5M sentence pairs,respectively. Each sentence was encoded into a sequence of sub-wordunits obtained by first tokenizing the sentence with the Mosestokenizer, then splitting tokens into sub-word units (also known as“word pieces”).

A shared vocabulary of 32K sub-word units is used for each source-targetlanguage pair. No further manual or rule-based post processing of theoutput was performed beyond combining the sub-word units to generate thetargets. Newstest 2014 includes the results, which serves as the testset. A combination of newstest 2012 and newstest 2013 is used forvalidation.

To evaluate the models, the BLEU metric is computed on tokenized,true-case output. For each training run, the model is evaluated every 30minutes on the development set. Once the model converges, the bestwindow is determined based on the average development-set BLEU scoreover 21 consecutive evaluations. The mean test score and standarddeviation are reported over the selected window. The model architecturescan be compared based on their mean performance after convergence ratherthan individual checkpoint evaluations, as the latter can be quite noisyfor some models.

To enable a fair comparison of architectures, the same pre-processingand evaluation methodology is used for each experiment. The checkpointaveraging (exponential moving averages of parameters) or checkpointensembles is refrained from being used to focus on evaluating theperformance of individual models.

The newly proposed RNMT+ model architecture is shown in FIG. 1. Here,the key architectural choices that are different between the RNMT+ modeland the GNMT model are highlighted. There are 6 bidirectional LSTMlayers in the encoder instead of 1 bidirectional LSTM layer followed by7 unidirectional layers as in GNMT. For each bidirectional layer, theoutputs of the forward layer and the backward layer are concatenatedbefore being fed into the next layer. The decoder network consists of 8unidirectional LSTM layers similar to the GNMT model. Residualconnections are added to the third layer and above for both the encoderand decoder. Inspired by the Transformer model, per-gate layernormalization is applied within each LSTM cell. The empirical resultsshow that layer normalization greatly stabilizes training. Nonon-linearity is applied to the LSTM output. A projection layer is addedto the encoder final output. Additional projection aims to reduce thedimensionality of the encoder output representations to match thedecoder stack dimension. Multi-head additive attention is used insteadof the single-head attention in the GNMT model. Similar to GNMT, thebottom decoder layer and the final encoder layer output is used afterprojection for obtaining the recurrent attention context. In addition tofeeding the attention context to all decoder LSTM layers, it is fed tothe softmax. This is important for both the quality of the models withmulti-head attention and the stability of the training process.

Since the encoder network in RNMT+ consists solely of bi-directionalLSTM layers, model parallelism is not used during training. Theresulting longer per-step time is compensated for with increased dataparallelism (more model replicas), so that the overall time to reachconvergence of the RNMT+ model is still comparable to that of GNMT.

The following regularization techniques are applied during training:dropout, label smoothing, and weight decay.

Dropout is applied to both embedding layers and each LSTM layer outputbefore it is added to the next layer's input. Attention dropout is alsoapplied.

Uniform label smoothing is used with an uncertainty=0.1. Label smoothingwas shown to have a positive impact on both Transformer and RNMT+models, especially in the case of RNMT+ with multi-head attention. It isbeneficial to use a larger beam size (e.g. 16, 20, etc.) during decodingwhen models are trained with label smoothing.

Here, t is the current step, n is the number of concurrent modelreplicas used in training, p is the number of warmup steps, s is thestart step of the exponential decay, and e is the end step of the decay.First, the learning rate is increased linearly during the number ofwarmup steps, keep it a constant until the decay start step s, thenexponentially decay until the decay end step e, and keep it at 5*10⁻⁵after the decay ends. This learning rate schedule is motivated by asimilar schedule that was successfully applied in training the Resnet-50model with a very large batch size.

In contrast to the asynchronous training used for GNMT, the RNMT+ modelscan be trained with synchronous training. The empirical results suggestthat when hyper-parameters are tuned properly, synchronous trainingoften leads to improved convergence speed and superior model quality.

To further stabilize training, adaptive gradient clipping is also used.A training step is completely disregarded if an anomaly in the gradientnorm value is detected, which is usually an indication of an imminentgradient explosion. More specifically, a moving average is monitored anda moving standard deviation of the log of the gradient norm values, anda step is aborted if the norm of the gradient exceeds four standarddeviations of the moving average.

In this section, the results of RNMT+ are compared with the results ofthe ConvS2S and Transformer.

For RNMT+, a sentence-level cross-entropy loss is used. Each trainingbatch contained 4096 sentence pairs (4096 source sequences and 4096target sequences). For ConvS2S and Transformer models, token-levelcross-entropy loss is used. Each training batch contained 65536 sourcetokens and 65536 target tokens. For the GNMT baselines on both tasks,the largest BLEU score reported is cited without reinforcement learning.

Table 1 shows results on the WMT′14 English to French task. Both theTransformer Big model and RNMT+ outperform GNMT and ConvS2S by about 2BLEU points. RNMT+ is slightly better than the Transformer Big model interms of its mean BLEU score. RNMT+ also yields a much lower standarddeviation, and hence, observed much less fluctuation in the trainingcurve. It takes approximately 3 days for the Transformer Base model toconverge, while both RNMT+ and the Transformer Big model require about 5days to converge. Although the batching schemes are quite differentbetween the Transformer Big and the RNMT+ model, they have processedabout the same amount of training samples upon convergence. The numbersbefore and after ‘±’ are the mean and standard deviation of test BLEUscore over an evaluation window.

TABLE 1 Results on WMT14 English to French. Training Model Test BLEUEpochs Time GNMT 38.95 — — ConvS2S⁷ 39.49 ± 0.11 62.2 438 h Trans. Base39.43 ± 0.17 20.7  90 h Trans. Big⁸ 40.73 ± 0.19 8.3 120 h RNMT+ 41.00 ±0.05 8.5 120 h

Table 2 shows our results on the WMT′14 English to German task. TheTransformer Base model improves over GNMT and ConvS2S by more than 2BLEU points while the Big model improves by over 3 BLEU points. RNMT+further outperforms the Transformer Big model and establishes a newstate of the art with an averaged value of 28.49. In this case, RNMT+converged slightly faster than the Transformer Big model and maintainedmuch more stable performance after convergence with a very smallstandard deviation, which is similar to the observance on the English toFrench task.

TABLE 2 Training Model Test BLEU Epochs Time GNMT 24.67 — — ConvS2S25.01 ± 0.17 38 20 h Trans. Base 27.26 ± 0.15 38 17 h Trans. Big 27.94 ±0.18 26.9 48 h RNMT+ 28.49 ± 0.05 24.6 40 h

Table 3 summarizes training performance and model statistics. TheTransformer Base model is the fastest model in terms of training speed.RNMT+ is slower to train than the Transformer Big model on a per-GPUbasis. However, since the RNMT+ model is quite stable, the lower per-GPUthroughput is offset with higher concurrency by increasing the number ofmodel replicas, and hence the overall time to convergence was not sloweddown much. The number of floating point operations (FLOPs) was alsocomputed in the model's forward path as well as the number of totalparameters for all architectures (see Table 3). RNMT+ requires fewerFLOPs than the Transformer Big model, even though both models have acomparable number of parameters. Examples are normalized by the numberof GPUs used in the training job. FLOPs are computed assuming thatsource and target sequence length are both 50.

TABLE 3 Performance comparison. Model Examples/s FLOPs ParametersConvS2S 80 15.7B 263.4M Trans. Base 160 6.2B 93.3M Trans. Big 50 31.2B375.4M RNMT+ 30 28.1B 378.9M

In this section, the importance of four main techniques for both theRNMT+ and the Transformer Big models are evaluated. These techniques areuniversally applicable across different model architectures, and shouldalways be employed by NMT practitioners for best performance.

The best RNMT+ and TransformerBig models are used and removing each oneof these techniques occurs independently. By doing this, two thingsabout each technique can be learned: (1) How much does it affect themodel performance? (2) How useful is it for stable training of othertechniques and hence the final model?

Table 4 shows ablation results of RNMT+ and the Transformer Big model onWMT′14 English to French. Average BLEU scores on the test set arereported. An asterisk ‘*’ indicates an unstable training run (traininghalts due to non-finite elements).

TABLE 4 Model RNMT+ Trans. Big Baseline 41.00 40.73 Label Smoothing40.33 40.49 Multi-head Attention 40.44 39.83 Layer Norm. * * Sync.Training 39.68 *

From Table 4, the various conclusions about the four techniques can bedrawn. Label smoothing improves both models, leading to an averageincrease of 0.7 BLEU for RNMT+ and 0.2 BLEU for Transformer Big models.Multi-head attention contributes significantly to the quality of bothmodels, resulting in an average increase of 0.6 BLEU for RNMT+ and 0.9BLEU for Transformer Big models. Of the factors shown in Table 4, layernormalization is most significant to stabilize the training process ofeither model, especially when multi-head attention is used. Removinglayer normalization results in unstable training runs for both models.Since by design, one technique can be removed at a time in our ablationexperiments, along with an inability to quantify how much layernormalization helped in either case. To be able to successfully train amodel without layer normalization, other parts of the model would haveto be adjusted and hyper-parameters would have to be returned.

Removing synchronous training has different effects on RNMT+ andTransformer. For RNMT+, it results in a significant quality drop, whilefor the Transformer Big model, it causes the model to become unstable.Also, synchronous training is only successful when coupled with atailored learning rate schedule that has a warmup stage at the beginning(see Eq. 1 for RNMT+ and Eq. 2 for Transformer). For RNMT+, removingthis warmup stage during synchronous training causes the model to becomeunstable.

In this section, hybrid architectures are explored that shed some lighton the salient behavior of each model family. These hybrid modelsoutperform the individual architectures on both benchmark datasets andprovide a better understanding of the capabilities and limitations ofeach model family.

In an encoder-decoder architecture, a natural assumption is that therole of an encoder is to build feature representations that can bestencode the meaning of the source sequence, while a decoder should beable to process and interpret the representations from the encoder and,at the same time, track the current target history. Decoding isinherently auto-regressive, and keeping track of the state informationshould therefore be intuitively beneficial for conditional generation.

Which family of encoders is more suitable to extract richrepresentations from a given input sequence is studied, and which familyof decoders can make the best of such rich representations. First,combining the encoder and decoder from different model families. Sinceit takes a significant amount of time for a ConvS2S model to converge,and because the final translation quality was not on par with the othermodels, two types of hybrids are focused on, a Transformer encoder withRNMT+ decoder and RNMT+ encoder with Transformer decoder.

TABLE 5 Results for encoder-decoder hybrids. Encoder Decoder En→Fr TestBLEU Trans. Big Trans. Big 40.73 ± 0.19 RNMT+ RNMT+ 41.00 ± 0.05 Trans.Big RNMT+ 41.12 ± 0.16 RNMT+ Trans. Big 39.92 ± 0.21

From Table 5, it is clear that the Transformer encoder is better atencoding or feature extraction than the RNMT+ encoder, whereas RNMT+ isbetter at decoding or conditional language modeling, confirming anintuition that a stateful decoder is beneficial for conditional languagegeneration.

The features extracted by an encoder can be further enhanced byincorporating additional information. For example, the combination oftransformer layers with RNMT+ layers in the same encoder block buildseven richer feature representations. RNMT+ decoders are used in thefollowing architectures, since stateful decoders have often shown betterperformance (see Table 5).

Two mixing schemes for combining RNMT+ and Transformer encoder layersare discussed below, the cascaded encoder arrangement and themulti-column encoder arrangement.

As illustrated in FIG. 4, the cascaded encoder aims at combining therepresentational power of RNNs and self-attention. The idea is to enricha set of stateful representations by cascading a feature extractor witha focus on vertical mapping. Our best performing cascaded encoderinvolves fine tuning transformer layers stacked on top of a pre-trainedfrozen RNMT+ encoder. Using a pre-trained encoder avoids optimizationdifficulties while significantly enhancing encoder capacity. As shown inTable 6 below, the cascaded encoder improves over the Transformerencoder by more than 0.5 BLEU points on the WMT′14 English to Frenchtask. This suggests that the Transformer encoder is able to extractricher representations if the input is augmented with sequentialcontext.

As illustrated in FIG. 5, a multi-column encoder merges the outputs ofseveral independent encoders into a single combined representation.Unlike a cascaded encoder, the multi-column encoder allows for furtherinvestigation as to whether an RNMT+ decoder can distinguish informationreceived from two different channels and benefit from its combination. Acrucial operation in a multi-column encoder is therefore how differentsources of information are merged into a unified representation. In somesituations, the best multi-column encoder performs a simpleconcatenation of individual column outputs.

The model details and hyper-parameters of the above two encoders aredescribed below. As shown in Table 6, the multi-column encoder followedby an RNMT+ decoder achieves better results than the Transformer and theRNMT model on both WMT′14 benchmark tasks.

In this work, the efficacy of several architectural and trainingtechniques proposed in recent studies on seq2seq models for NMT has beenexplored here. Many of these techniques are broadly applicable tomultiple model architectures. Applying these new techniques to RNMTmodels yields RNMT+, an enhanced RNMT model that significantlyoutperforms the three fundamental architectures on WMT′14 English toFrench and English to German tasks. Several hybrid models are furtherdeveloped by combining encoders and decoders from the Transformer andRNMT+ models, and empirically demonstrated the superiority of theTransformer encoder and the RNMT+ decoder in comparison with theircounterparts. The encoder's architecture is enhanced by horizontally andvertically mixing components borrowed from these architectures, leadingto hybrid architectures that obtain further improvements over RNMT+.

TABLE 6 Results for hybrids with cascaded encoder and multi-columnencoder. Model En→Fr BLEU En→De BLEU Trans. Big 40.73 ± 0.19 27.94 ±0.18 RNMT+ 41.00 ± 0.05 28.59 ± 0.05 Cascaded 41.67 ± 0.11 28.62 ± 0.06MultiCol 41.66 ± 0.11 28.84 ± 0.06

For the WMT′14 English to German task, both the encoder and decoder have15 layers, with 512 hidden units in the first ten layers, 768 units inthe subsequent three layers and 2048 units in the final two layers. Thefirst 13 layers use kernel width 3 and the final two layers use kernelwidth 1. For the WMT′14 English to French task, both the encoder anddecoder have 14 layers, with 512 hidden units in the first five layers,768 units in the subsequent four layers, 1024 units in the next threelayers, and 2048 units and 4096 units in the final two layers. The first12 layers use kernel width 3 and the final two layers use kernelwidth 1. The ConvS2S models are trained with synchronous training using32 GPUs.

Both the encoder and the decoder have 6 Transformer layers. Transformerbase model has model dimension 512, hidden dimension 2048, and 8attention heads. The Transformer Big model uses model dimension 1024,hidden dimension 8192, and 16 attention heads. The dropout inTransformer models are grouped into four types: input dropout—dropoutapplied to the sum of token embeddings and position encodings; residualdropout—dropout applied to the output of each sublayer before added tothe sublayer input; relu dropout—dropout applied to the inner layeroutput after ReLU activation in each feed-forward sub-layer; and,attention dropout—dropout applied to attention weight in each attentionsub-layer. All Transformer models use the following learning rateschedule:

$\begin{matrix}{{lr} = {\frac{r_{0}}{\sqrt{d_{model}}}*\min\mspace{11mu}{\left( {\frac{t + 1}{p\sqrt{p}},\frac{1}{\sqrt{\left( {t + 1} \right)}}} \right).}}} & (2)\end{matrix}$

where t is the current step, p is the number of warmup steps, d_(model)is the model dimension, and r₀ is a constant to adjust the magnitude ofthe learning rate.

On WMT′14 English to German, the Transformer Base model employs all fourtypes of dropout with dropout_probs=0.1. r₀=2.0 and p=8000 is used inthe learning rate schedule. For the Transformer Big model, only residualdropout and input dropout are applied, both with dropout_probs=0.3.r₀=3.0 and p=40000 are used in the learning rate schedule.

Both Transformer base model and big model are trained with synchronoustraining using 16 GPUs.

RNMT+ has 1024 LSTM nodes in all encoder and decoder layers. The inputembedding dimension is 1024 as well. The encoder final projection layerprojects the last bidirectional layer output from dimension 2048 to1024. 4 attention heads are used in the multi-head additive attention.Label smoothing is applied with an uncertainty=0.1. FIG. 3 illustratesour learning rate schedule defined in Eq. 1.

For both encoder-decoder hybrids, i.e., Transformer Big encoder withRNMT+ decoder and RNMT+ encoder with Transformer Big decoder, the exactsame model hyper-parameters are used as in the Transformer Big and RNMT+models described previously.

In this hybrid, a transformer encoder is stacked on top of the RNMT+encoder. During experimentation, a pre-trained RNMT+ encoder is used,including the projection layer, as previously described. The outputs ofthe RNMT+ encoder are layer normalized and fed into a transformerencoder. This structure is illustrated in FIG. 2a . The transformerencoder is identical to the one described previously per the conditionaltransformation-based NMT models except for the different number oflayers. Another setup includes 4 Transformer layers stacked on top of apre-trained RNMT+ encoder with 6 layers. To speed up convergence, thegradient updates were frozen in the pre-trained RNMT+ encoder. Thisenables increasing the encoder capacity significantly, while avoidingoptimization issues encountered in non-frozen variants of the hybrid. Asan additional benefit, this enables training the model on P100s withoutthe need for model parallelism.

Note that this specific layout allows for dropping hand-craftedsinusoidal positional embeddings (since positional information isalready captured by the underlying RNNs).

A simple concatenation as the merger-operator without fine-tuning anyother model hyper-parameters. After concatenation, the combinedrepresentation is projected down to the decoder dimension with alayer-normalized affine transformation. Although only two columns areillustrated here, there is no practical restriction on the total numberof columns that this hybrid can combine. By combining multiple encoderrepresentations, the network may capture different factors of variationsin the input sequence.

Similar to the Cascaded-RNMT+ hybrid, pre-trained encoders are used thatare borrowed from an RNMT+ model (a pre-trained RNMT+ encoder is used asthe first column) and an Encoder-Decoder hybrid model with Transformerencoder and RNMT+ decoder (the pre-trained Transformer encoder is used).Multi-column encoder with RNMT+ decoder is trained using 16 GPUs in asynchronous training setup. The simple concatenation operation is usedas the merger-operator, and after concatenation, the combinedrepresentation is projected down the decoder dimension with a simplelayer-normalized affine transformation. Additionally, for the sake ofstability and trainability, each column output should be first mapped toa space where the representation ranges are compatible, e.g., RNMT+encoder output has no limitation on its range, but a Transformer Encoderoutput range is constrained by the final layernormalization applied tothe entire Transformer encoder body. Therefore, a layer normalization isapplied to the RNMT+ encoder outputs to match the ranges of individualencoders.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non transitory program carrier for execution by, or to controlthe operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on an artificiallygenerated propagated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal that is generated to encodeinformation for transmission to suitable receiver apparatus forexecution by a data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them. The computer storage medium is not, however, apropagated signal.

FIG. 7 shows an example of a computing device 700 and a mobile computingdevice 750 that can be used to implement the techniques described here.The computing device 700 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The mobile computing device 750 is intended torepresent various forms of mobile devices, such as personal digitalassistants, cellular telephones, smart-phones, and other similarcomputing devices. The components shown here, their connections andrelationships, and their functions, are meant to be examples only, andare not meant to be limiting.

The computing device 700 includes a processor 702, a memory 704, astorage device 706, a high-speed interface 708 connecting to the memory704 and multiple high-speed expansion ports 710, and a low-speedinterface 712 connecting to a low-speed expansion port 714 and thestorage device 706. Each of the processor 702, the memory 704, thestorage device 706, the high-speed interface 708, the high-speedexpansion ports 710, and the low-speed interface 712, are interconnectedusing various busses, and may be mounted on a common motherboard or inother manners as appropriate. The processor 702 can process instructionsfor execution within the computing device 700, including instructionsstored in the memory 704 or on the storage device 706 to displaygraphical information for a GUI on an external input/output device, suchas a display 716 coupled to the high-speed interface 708. In otherimplementations, multiple processors and/or multiple buses may be used,as appropriate, along with multiple memories and types of memory. Also,multiple computing devices may be connected, with each device providingportions of the necessary operations (e.g., as a server bank, a group ofblade servers, or a multi-processor system).

The memory 704 stores information within the computing device 700. Insome implementations, the memory 704 is a volatile memory unit or units.In some implementations, the memory 704 is a non-volatile memory unit orunits. The memory 704 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 706 is capable of providing mass storage for thecomputing device 700. In some implementations, the storage device 706may be or contain a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. Instructions can be stored in an information carrier.The instructions, when executed by one or more processing devices (forexample, processor 702), perform one or more methods, such as thosedescribed above. The instructions can also be stored by one or morestorage devices such as computer- or machine-readable mediums (forexample, the memory 704, the storage device 706, or memory on theprocessor 702).

The high-speed interface 708 manages bandwidth-intensive operations forthe computing device 700, while the low-speed interface 712 manageslower bandwidth-intensive operations. Such allocation of functions is anexample only. In some implementations, the high-speed interface 708 iscoupled to the memory 704, the display 716 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 710,which may accept various expansion cards (not shown). In theimplementation, the low-speed interface 712 is coupled to the storagedevice 706 and the low-speed expansion port 714. The low-speed expansionport 714, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 700 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 720, or multiple times in a group of such servers. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 722. It may also be implemented as part of a rack server system724. Alternatively, components from the computing device 700 may becombined with other components in a mobile device (not shown), such as amobile computing device 750. Each of such devices may contain one ormore of the computing device 700 and the mobile computing device 750,and an entire system may be made up of multiple computing devicescommunicating with each other.

The mobile computing device 750 includes a processor 752, a memory 764,an input/output device such as a display 754, a communication interface766, and a transceiver 768, among other components. The mobile computingdevice 750 may also be provided with a storage device, such as amicro-drive or other device, to provide additional storage. Each of theprocessor 752, the memory 764, the display 754, the communicationinterface 766, and the transceiver 768, are interconnected using variousbuses, and several of the components may be mounted on a commonmotherboard or in other manners as appropriate.

The processor 752 can execute instructions within the mobile computingdevice 750, including instructions stored in the memory 764. Theprocessor 752 may be implemented as a chipset of chips that includeseparate and multiple analog and digital processors. The processor 752may provide, for example, for coordination of the other components ofthe mobile computing device 750, such as control of user interfaces,applications run by the mobile computing device 750, and wirelesscommunication by the mobile computing device 750.

The processor 752 may communicate with a user through a controlinterface 758 and a display interface 756 coupled to the display 754.The display 754 may be, for example, a TFT (Thin-Film-Transistor LiquidCrystal Display) display or an OLED (Organic Light Emitting Diode)display, or other appropriate display technology. The display interface756 may comprise appropriate circuitry for driving the display 754 topresent graphical and other information to a user. The control interface758 may receive commands from a user and convert them for submission tothe processor 752. In addition, an external interface 762 may providecommunication with the processor 752, so as to enable near areacommunication of the mobile computing device 750 with other devices. Theexternal interface 762 may provide, for example, for wired communicationin some implementations, or for wireless communication in otherimplementations, and multiple interfaces may also be used.

The memory 764 stores information within the mobile computing device750. The memory 764 can be implemented as one or more of acomputer-readable medium or media, a volatile memory unit or units, or anon-volatile memory unit or units. An expansion memory 774 may also beprovided and connected to the mobile computing device 750 through anexpansion interface 772, which may include, for example, a SIMM (SingleIn Line Memory Module) card interface. The expansion memory 774 mayprovide extra storage space for the mobile computing device 750, or mayalso store applications or other information for the mobile computingdevice 750. Specifically, the expansion memory 774 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, theexpansion memory 774 may be provide as a security module for the mobilecomputing device 750, and may be programmed with instructions thatpermit secure use of the mobile computing device 750. In addition,secure applications may be provided via the SIMM cards, along withadditional information, such as placing identifying information on theSIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory(non-volatile random access memory), as discussed below. In someimplementations, instructions are stored in an information carrier, suchthat the instructions, when executed by one or more processing devices(for example, processor 752), perform one or more methods, such as thosedescribed above. The instructions can also be stored by one or morestorage devices, such as one or more computer- or machine-readablemediums (for example, the memory 764, the expansion memory 774, ormemory on the processor 752). In some implementations, the instructionscan be received in a propagated signal, for example, over thetransceiver 768 or the external interface 762.

The mobile computing device 750 may communicate wirelessly through thecommunication interface 766, which may include digital signal processingcircuitry where necessary. The communication interface 766 may providefor communications under various modes or protocols, such as GSM voicecalls (Global System for Mobile communications), SMS (Short MessageService), EMS (Enhanced Messaging Service), or MMS messaging (MultimediaMessaging Service), CDMA (code division multiple access), TDMA (timedivision multiple access), PDC (Personal Digital Cellular), WCDMA(Wideband Code Division Multiple Access), CDMA2000, or GPRS (GeneralPacket Radio Service), among others. Such communication may occur, forexample, through the transceiver 768 using a radio-frequency. Inaddition, short-range communication may occur, such as using aBluetooth, WiFi, or other such transceiver (not shown). In addition, aGPS (Global Positioning System) receiver module 770 may provideadditional navigation- and location-related wireless data to the mobilecomputing device 750, which may be used as appropriate by applicationsrunning on the mobile computing device 750.

The mobile computing device 750 may also communicate audibly using anaudio codec 760, which may receive spoken information from a user andconvert it to usable digital information. The audio codec 760 maylikewise generate audible sound for a user, such as through a speaker,e.g., in a handset of the mobile computing device 750. Such sound mayinclude sound from voice telephone calls, may include recorded sound(e.g., voice messages, music files, etc.) and may also include soundgenerated by applications operating on the mobile computing device 750.

The mobile computing device 750 may be implemented in a number ofdifferent forms, as shown in the figure. For example, it may beimplemented as a cellular telephone 780. It may also be implemented aspart of a smart-phone 782, personal digital assistant, or other similarmobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term machine-readable signal refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Although a few implementations have been described in detail above,other modifications are possible. For example, while a clientapplication is described as accessing the delegate(s), in otherimplementations the delegate(s) may be employed by other applicationsimplemented by one or more processors, such as an application executingon one or more servers. In addition, the logic flows depicted in thefigures do not require the particular order shown, or sequential order,to achieve desirable results. In addition, other actions may beprovided, or actions may be eliminated, from the described flows, andother components may be added to, or removed from, the describedsystems. Accordingly, other implementations are within the scope of thefollowing claims.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A computer-implemented method for performingmachine translation of text from a first language to a second language,the method comprising: generating, by one or more processors, a set ofencoding vectors from a series of feature vectors representingcharacteristics of a text segment in the first language, by processingthe feature vectors with an encoder neural network comprising a set ofbidirectional recurrent neural network layers, each encoding vector ofthe set having a predetermined number of values; generating, by the oneor more processors, multiple context vectors for each encoding vectorbased on multiple sets of parameters, the multiple sets of parametersbeing respectively used to generate the context vectors from differentsubsets of each encoding vector; generating, by the one or moreprocessors, a sequence of output vectors using a decoder neural networkthat receives the context vectors, the decoder neural network comprisinga recurrent neural network, the output vectors representingdistributions over language elements of the second language; anddetermining, by the one or more processors, a translation of the textsegment into the second language based on the sequence of outputvectors.
 2. The method of claim 1, further comprising: providing thetranslation to a client devices in response to a translation request. 3.The method of claim 1, wherein the recurrent neural network in thedecoder neural network does not perform self-attention.
 4. The method ofclaim 1, wherein the set of bidirectional recurrent neural networklayers of the encoder neural network comprise long short-term memory(LSTM) layers.
 5. (canceled)
 6. The method of claim 1, wherein theparameters are weighting values, and the different sets of theparameters are applicable to different non-overlapping continuous chunksof the encoding vectors. 7-8. (canceled)
 9. The method of claim 1,wherein the decoder neural network is configured to receive the contextvectors, concatenated together, at a softmax layer providing output ofthe decoder neural network.
 10. (canceled)
 11. The method of claim 1,wherein two or more of the language elements of the second languagecomprise characters, word pieces, words, or phrases.
 12. (canceled) 13.The method of claim 1, wherein the encoder neural network and thedecoder neural network include a normalization layer between eachrecurrent hidden neural network layer.
 14. The method of claim 1,wherein at least one of the encoder neural network and the decoderneural network have been trained using synchronous training.
 15. Themethod of claim 1, wherein at least one of the encoder neural networkand the decoder neural network have been trained using a learning ratethat increases gradually during training.
 16. The method of claim 1,wherein at least one of the encoder neural network and the decoderneural network have been trained using label smoothing that introducesvariability into target labels.
 17. (canceled)
 18. The method of claim1, wherein: the encoder neural network comprises a first encoder moduleand a second encoder module, the first encoder module and the secondencoder module having different neural network topologies; the firstencoder module uses a transformer layer structure with layers that eachinclude (i) a self-attention network sub-layer and (ii) a feed-forwardnetwork sub-layer; and the second encoder module includes a series ofbidirectional recurrent neural network layers each providingnormalization before processing by a next recurrent layer.
 19. A systemcomprising: one or more processors; and one or more data storage devicesstoring instructions that, when executed by the one or more processors,cause the one or more processors to perform operations comprising:generating a set of encoding vectors from a series of feature vectorsrepresenting characteristics of a text segment in a first language byprocessing the series of feature vectors with an encoder neural networkcomprising a set of bidirectional recurrent neural network layers, eachencoding vector of the set having a predetermined number of values;generating multiple context vectors for each encoding vector based onmultiple sets of parameters, the multiple sets of parameters beingrespectively used to generate the context vectors from different subsetsof each encoding vector; generating a sequence of output vectors using adecoder neural network that receives the context vectors, the decoderneural network comprising a recurrent neural network, the output vectorsrepresenting distributions over language elements of a second language;and determining a translation of the text segment into the secondlanguage based on the sequence of output vectors.
 20. (canceled)
 21. Acomputer-implemented method for performing machine translation of textfrom a first language to a second language, the method comprising:obtaining, by a machine translation module, a source embeddingassociated with a text segment in the first language, the sourceembedding having a selected dimensionality; processing, with an encoderneural network of the machine translation module, the source embeddingto obtain a set of encoding vectors, the encoder neural networkcomprising a set of bidirectional recurrent neural network layersincluding a plurality of forward-propagating layers and a plurality ofbackward-propagating layers; normalizing, by the machine translationmodule, the set of encoding vectors; processing, by a transformerencoder of the machine translation module, the normalized set ofencoding vectors to obtain transformed data; generating output vectorsusing a decoder neural network from the transformed data, the decoderneural network comprising a recurrent neural network, the output vectorsrepresenting distributions over language elements of the secondlanguage; and determining, by the machine translation module, atranslation of the text segment into the second language based on theoutput vectors.
 22. The method of claim 21, further comprising providingthe translation to a client device in response to a translation request.23. The method of claim 21, wherein the recurrent neural network of thedecoder neural network does not perform self-attention.
 24. The methodof claim 21, wherein feature extraction performed by the encoder neuralnetwork is cascaded to the transformer encoder.
 25. The method of claim21, wherein the set of bidirectional recurrent neural network layers ofthe encoder neural network comprise long short-term memory (LSTM)layers.
 26. The method of claim 21, wherein processing the normalizedset of encoding vectors includes performing multi-headed attention. 27.The method of claim 21, wherein: the transformer encoder comprises afirst encoder module and a second encoder module, the first encodermodule and the second encoder module having different neural networktopologies; the first encoder module uses a transformer layer structurewith layers that each include (i) a self-attention network sub-layer and(ii) a feed-forward network sub-layer; and the second encoder moduleincludes a series of bidirectional recurrent neural network layers eachproviding normalization before processing by a next recurrent layer.